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Scalable  Coordination  Architectures  for  Deeply 
Distributed  Systems 
Final  Report 

29  June  1999  -  31  March  2003 
1 .  Introduction 

This  report  summarizes  the  progress  of  the  entire  SCADDS  project  at  ISI  from 
June  29,  1999  to  March  31,  2003.  Over  this  period,  the  SCADDS  project  was 
overseen  by  the  principal  investigators:  Deborah  Estrin,  John  Heidemann,  and 
Ramesh  Govindan. 

The  SCADDS  project  explored  and  demonstrated  scalable  coordination 
mechanisms  for  deeply  distributed  and  dynamic  systems.  Nodes  in  these  systems 
were  heterogeneous,  having  a  range  of  sensing,  actuation  and  communication 
capabilities.  These  systems  raised  many  challenges  for  distributed  system  and 
network  design.  The  first  was  a  shift  from  node-centric  to  data-centric  network 
architecture.  Both  scalability  and  long  lifetime  called  for  extensive  processing  of 
data  within  and  among  the  nodes  of  the  sensor  network.  Rather  than  streaming 
all  sensor  readings  back  to  a  central  site  for  processing,  nodes  autonomously 
exchanged  data,  filtered  out  uninteresting  events,  and  identified  patterns  of 
interest.  The  second  challenge  was  to  build  systems  that  were  truly  self- 
configuring;  able  to  adapt  efficiently  to  ad  hoc  deployment  and  both 
environmental  and  network  dynamics.  This  paradigm  shift  required  new  network 
architecture.  The  SCADDS  project  investigated  an  approach  called  directed 
diffusion.  The  report  is  divided  into  the  following  subject  areas: 

Directed  Diffusion  Architecture 

Diffusion  Architecture: 

Diffusion  is  a  tunable  algorithm  for  information  dissemination  in  wireless  sensor 
networks.  Rapid  deployment  of  large  numbers  of  sensors  in  dynamic  and 
potentially  hostile  environments  presents  a  challenge  to  existing  networking 
techniques,  in  terms  of  scalability,  robustness,  and  adaptability.  The  SCADDS 
project  explored  the  use  of  localized  algorithms  as  a  building  block  for  such 
dynamic  data-dissemination  systems. 

Tiny  Diffusion 

Full  diffusion  is  designed  for  32-bit  computer.  Tiny  Diffusion  is  a  subset  of  Full 
Diffusion  that  runs  on  Berkley  motes. 

Geographical  and  Energy  Aware  Routing  (GEAR) 

Geographic  and  Energy  Aware  Routing  (GEAR)  is  an  extension  to  diffusion  that 
uses  energy  aware  and  geographically  informed  neighbor  selection  to  route  a 
packet  towards  the  target  region.  This  strategy  attempts  to  balance  node  energy 
consumption  and  thereby  increase  network  lifetime.  Within  a  region,  it  uses  a 
recursive  geographic  forwarding  technique  or  a  restricted  flooding  algorithm  to 
disseminate  the  packet. 
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Sensor-MAC  (S-MAC) 


The  S-MAC  is  an  energy-efficient  MAC  protocol  explicitly  designed  for  wireless 
sensor  networks.  Its  major  goal  is  to  reduce  energy  consumption  while 
maintaining  good  properties  such  as  collision  avoidance,  scalability,  and  self¬ 
organization.  The  protocol  reduces  energy  consumption  through  the  following 
approaches:  reduce  idle  listening,  avoid  collisions,  avoid  overhearing  and  reduce 
control  overhead. 

Application  Techniques 

Complementing  network-level  advances,  the  SCADDS  project  explored  several 
application-level  techniques  to  improve  sensor  net  performance. 

Clustering 

Clustering  can  contribute  to  more  scalable  system  behavior  as  number  of  sensors 
increase,  improved  robustness,  and  efficient  resource  utilization  for  many 
distributed  sensor  coordination  tasks.  However,  self-configuring  techniques 
required  to  organize  sensors  into  clusters  (since  manual  configuration  of  a  large 
number  of  sensors  is  not  feasible)  can  consume  significant  resources  that  need  to 
amortized  over  the  gains  in  application  function.  The  SCADDS  project 
investigated  this  trade-off  with  energy  as  the  primary  resource  constraint  in  the 
context  of  an  object  tracking  sensor  network  application.  In  this  application, 
sensors  monitor  location  and  status  of  various  tagged  objects  (e.g.,  projectors, 
cameras).  Queries  from  users  about  these  tagged  objects  were  efficiently  resolved 
through  a  hierarchical  scheme. 

Aggregation 

Aggregation  can  be  expressed  as  local  transformation  of  data,  to  reduce  overall 
power  consumption.  Hence,  it  might  take  various  forms:  from  compressing  data, 
to  merging  data  temporally  or  spatially  (i.e.  data  coming  from  different  neighbors), 
to  only  forwarding  deductions  on  the  basis  of  received  data. 

Topology  Control 

Topology  control  protocols  extend  network  lifetime  by  periodically  turning  node  or 
node  radios  off.  The  sleeping  time  of  radio  interface  can  be  adapted  according  to 
the  nodes’  density.  The  idea  uses  network  density  as  a  clue  to  put  some  nodes 
into  sleep  mode.  The  design  take  advantage  of  the  existing  ad  hoc  routing 
protocols  to  find  network  neighborhood  information,  and  adapts  to  the  node’s 
sleep  time  according  to  the  number  of  its  neighbors.  The  neighborhood 
information  is  stored  in  soft  state  so  that  the  algorithm  can  adapt  to  the  network 
dynamics.  The  document  explains  several  different  topological  control  protocols 
BECA,  AFECA,  GAF,  and  CEC,  both  in  simulation  and  experimentation. 

Infrastructure  and  Collaboration 

An  emphasis  as  part  of  SensIT  was  on  providing  research  and  tools  that  can  be 
used  by  other  researchers. 

SCADDS  was  very  active  at  integration  and  SensIT  field  tests  at  29  Palms. 
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Testbed  Development 

The  ISI  testbed  provides  an  experimentation  environment  to  study 
communication  protocols  in  a  large  network  with  different  topologies.  With 
feedback  from  the  well-instrumented  real-world  experiments,  one  can  iteratively 
improve  performance  of  protocols  and  algorithms  and  validate  simulation  results. 
Moreover,  playing  with  real  systems  can  lead  us  to  better  understanding  of  the 
design  challenges. 

Software 

The  project  developed  many  software  tools  over  the  course  of  this  project,  both  as 
research  itself  and  as  tools  to  support  the  research.  These  include:  directed 
diffusion  with  support  for  hardware  Sensoria,  WINSng  1  and  2,  Ethernet 
evaluation,  Radiometrix,  mote  radios,  extensions  in  aggregation  GEAR,  PUSH, 
and  nested  queries.  Diffusion  was  also  posted  to  run  inside  the  ns-2  simulator. 
The  project  also  developed  these  utilities:  Emlog  and  Parapin  Diffusion 
visualization. 

2.  Directed  Diffusion 

Directed  diffusion  [IntanagonwiwatOOa]  is  a  data-centric  data  dissemination 
protocol.  Data  generated  by  sensor  nodes  is  named  by  attribute-value  pairs.  A 
node  requests  data  by  sending  interests  for  named  data.  Data  matching  the 
interest  is  then  "drawn"  down  towards  that  node.  Intermediate  nodes  can  cache, 
or  transform  data,  and  may  direct  interests  based  on  previously  cached  data 
(Section  3). 

Directed  diffusion  is  significantly  different  from  IP-style  communication  where 
nodes  are  identified  by  their  end-points,  and  inter-node  communication  is  layered 
on  an  end-to-end  delivery  service  provided  within  the  network.  In  directed 
diffusion,  nodes  in  the  network  are  application-aware  as  we  allow  application- 
specific  code  to  run  in  the  network  and  assist  diffusion  in  processing  messages. 
This  allows  directed  diffusion  to  cache  and  process  data  in  the  network 
(aggregation),  decreasing  the  amount  of  end-to-end  traffic,  and  resulting  in  higher 
energy  savings.  We  show  that  using  directed  diffusion  one  can  realize  robust 
multi-path  delivery,  empirically  adapt  to  a  small  subset  of  network  paths,  and 
achieve  significant  energy  savings  when  intermediate  nodes  aggregate  responses 
to  queries  (Section  5). 

2.1  The  Publish /Subscribe  API  and  Data  Naming 

Directed  diffusion  uses  a  publish/subscribe-based  API.  To  receive  data,  users  or 
programs  subscribe  to  a  particular  set  of  attributes,  becoming  data  sinks.  A 
callback  function  is  then  invoked  whenever  relevant  data  arrives  at  the  node. 
Sensors  publish  data  that  they  have,  becoming  data  sources.  In  both  cases,  what 
data  is  provided  or  received  is  described  by  an  attribute-based  naming  scheme 
described  next.  It  is  the  job  of  the  diffusion  dissemination  algorithms  to  ensure 
that  data  is  communicated  efficiently  from  sources  to  sinks  across  a  multi-hop 
network.  In  general,  publishing  and  subscribing  sends  messages  across  the 
network.  The  exact  cost  of  these  operations  depends  on  which  diffusion  algorithm 
is  used. 
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To  allow  applications  to  influence  data  as  it  moves  through  the  network,  users 
can  create  filters  at  each  sensor  node  with  the  filter  APIs.  Filters  indicate  what 
messages  they  are  interested  in  by  attributes;  each  time  a  matching  message 
arrives  at  that  node  the  filter  is  allowed  to  inspect  and  alter  its  progress  in  any 
way.  Filters  can  suppress  messages,  change  where  they  are  sent  next,  or  even 
send  other  messages  in  response  to  one  (perhaps  triggering  further  sensors  to 
satisfy  a  query). 

Diffusion  uses  an  attribute-based  naming  scheme  to  associate  sources  and  sinks 
and  to  trigger  filters.  This  flexible  approach  to  naming  is  important  in  several  ways. 
First,  attribute-based  naming  is  consistent  with  the  publish/ subscribe  application- 
level  interface  (just  described)  and  many-to-many  communication.  Diffusion's 
naming  scheme  is  data-centric,  allowing  applications  to  focus  on  what  data  is 
desired  rather  than  on  individual  sensor  nodes.  The  approach  also  supports 
multiple  sources  and  sinks,  rather  than  simple  point-to-point  communication. 
Thus  applications  may  subscribe  to  "seismic  sensors  in  the  southeast  region" 
rather  seismic  sensors  #15  and  #35,  or  hosts  10.1.2.40  and  10.2.1.88. 

Second,  diffusion  attributes  provide  some  structure  to  a  message.  By  identifying 
separate  fields,  data  dissemination  algorithms  can  use  application  data  to 
influence  routing.  For  example,  application-specific,  geographic  information  can 
limit  where  diffusion  must  look  for  sensors.  In  addition,  treating  messages  as  sets 
of  attributes  simplifies  application  and  protocol  extensions  (a  need  also  suggested 
for  future  Internet-based  protocols). 

Finally,  attributes  serve  to  associate  messages  with  sources,  sinks,  and  filters  via 
matching.  If  the  attributes  in  a  sink's  subscription  match  those  of  source's 
publication,  diffusion  must  send  any  published  data  to  the  sink. 

2.2  Directed  Diffusion  Protocol  Family 

Publish/ subscribe  provides  an  application's  view  to  a  sensor  network,  and 
attribute-based  naming  a  detailed  way  to  specify  which  sources  and  sinks 
communicate.  The  "glue"  that  binds  the  two  are  the  directed  diffusion  algorithms 
for  data  dissemination.  In  a  traditional  network,  communication  is  effected  by 
routing,  usually  based  on  global  addresses  and  routing  metrics.  Instead,  we  use 
the  term  data  dissemination  to  emphasize  the  lack  of  global  addresses,  reliance 
on  local  rules,  and  the  use  of  application-specific  in-network  processing. 

The  original,  two-phase  directed  diffusion  uses  several  control  messages  to  realize 
our  publish/ subscribe  API:  sinks  send  interest  messages  to  find  sources,  sources 
use  exploratory  data  messages  to  find  sources,  and  positive  and  negative 
reinforcement  messages  select  or  prune  parts  of  the  path.  Early  work 
[IntanagonwiwatOOa]  identified  these  primitives,  described  the  concept  of 
diffusion,  and  evaluated  a  specific  algorithm  that  we  now  call  two-phase  pull 
diffusion.  We  found  this  algorithm  ideal  for  some  applications  but  as  our 
experience  with  sensor  networks  applications  grew,  we  found  two-phase  pull  a 
poor  match  for  other  classes  of  applications. 

We  see  diffusion  not  as  a  single  algorithm,  but  as  a  family  of  algorithms  built 
from  these  primitives.  Other  algorithms  provide  better  performance  for  some 


4 


applications.  We  have  recently  made  two  additions  to  the  diffusion  protocol 
family:  one-phase  push  and  one-phase  pull  [Heidemann03b]. 

Over  the  course  of  the  project,  SCADDS  researchers  developed  several 
verresionons  of  diffusion  with  different  performance  characteristics  and  evaluated 
them  through  analysis,  simulation,  and  experimentation.  See 
[Intanagonwiwat03a]  for  the  primary  analytic  results,  [IntanagonwiwatOOa]  and 
[Heidemann03b]  for  simulation  results  evaluating  the  basic  algorithm  and 
variants,  and  [HeidemannOlc]  for  experimental  results.  We  summarize  the  key 
results  below. 

2.3  Comparing  diffusion  with  alternatives 

Our  first  experiment  compares  diffusion  to  omniscient  multicast  and  flooding 
scheme  for  data  dissemination  in  networks.  Figure  6(a)  shows  the  average 
dissipated  energy  per  packet  as  a  function  of  network  size.  Omniscient  multicast 
dissipates  a  little  less  than  a  half  as  much  energy  per  packet  per  node  than 
flooding.  It  achieves  such  energy  efficiency  by  delivering  events  along  a  single 
path  from  each  source  to  every  sink.  Directed  diffusion  has  noticeably  better 
energy  efficiency  than  omniscient  multicast.  For  some  sensor  fields,  its  dissipated 
energy  is  only  60%  that  of  omniscient  multicast.  As  with  omniscient  multicast,  it 
also  achieves  significant  energy  savings  by  reducing  the  number  of  paths  over 
which  redundant  data  is  delivered.  In  addition,  diffusion  benefits  significantly 
from  in-network  aggregation.  In  our  experiments,  the  sources  deliver  identical 
location  estimates,  and  intermediate  nodes  suppress  duplicate  location 
estimates.  This  corresponds  to  the  situation  where  there  is,  for  example,  a  single 
vehicle  in  the  specified  region. 

Figure  (1)  plots  the  average  delay  observed  as  a  function  of  network  size. 
Directed  diffusion  has  a  delay  comparable  to  omniscient  multicast.  This  is 
encouraging.  To  a  first  approximation,  in  an  uncongested  sensor  network  and  in 
the  absence  of  obstructions,  the  shortest  path  is  also  the  lowest  delay  path. 
Thus,  our  reinforcement  rules  seem  to  be  finding  the  low  delay  paths.  However, 
the  delay  experienced  by  flooding  is  almost  an  order  of  magnitude  higher  them 
other  schemes.  This  is  an  artifact  of  the  MAC  layer:  to  avoid  broadcast  collisions, 
a  randomly  chosen  delay  is  imposed  on  all  MAC  broadcasts.  Flooding  uses  MAC 
broadcasts  exclusively.  Diffusion  only  uses  such  broadcasts  to  propagate  the 
initial  interests.  On  a  sensor  radio  that  employs  a  TDMA  MAC-layer,  we  might 
expect  flooding  to  exhibit  a  delay  comparable  to  the  other  schemes. 

In  summary,  directed  diffusion  exhibits  better  energy  dissipation  than 
omniscient  multicast  and  has  good  latency  properties. 
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Figure  1:  Modules  using  filter  architecture  with  several  variants  of  directed  diffusion 


2.4  Tiny  Diffusion 

TinyDiffusion  is  implemented  as  a  component  in  TinyOS  that  adds  3250B  code 
and  144B  of  data  (including  support  for  radio  and  a  photo  sensor),  so  the  entire 
system  runs  in  less  than  5.5KB  of  memory. 

TinyDiffusion  is  statically  configured  to  support  for  5  active  gradients  and  a 
cache  of  10  packets  of  the  2  relevant  bytes  per  packet.  Although  reduced  in  size, 
the  logical  header  format  is  compatible  with  that  of  the  full  diffusion 
implementation  and  SCADDS  is  implementing  software  to  gateway  between  the 
implementations.  Although  SCADDS  does  not  currently  provide  filters  in  tiny- 
diffusion,  they  are  an  essential  component  of  enabling  in-network  aggregation  in 
diffusion,  and  SCADDS  intends  to  add  them.  The  team  intends  to  leverage  on  the 
ability  to  reprogram  motes  over  the  air  to  program  filters  dynamically. 

Motes  and  tiny-diffusion  can  be  used  in  regions  where  there  is  need  for  dense 
sensor  distribution,  such  as  distributing  photo  sensors  in  a  room  to  detect 
change  in  light  or  temperature  sensors  for  fine  grained  sensing.  They  provide  the 
necessary  sensor  data  processing  capability,  with  the  ability  to  use  diffusion  to 
communicate  with  less  resource  constrained  nodes  (PC  /104-class  nodes  with  the 
full  system).  They  can  also  be  used  to  provide  additional  multi-hop  capability 
under  adverse  wireless  communication  conditions. 

We  thus  envisage  deployment  of  a  tiered  architecture  in  which  less  resource 
constrained  nodes  running  the  full  system  will  form  the  highest  tier,  and  act  as 
gateways  to  the  second  tier  comprising  motes  connected  to  low-power  sensors 
running  tiny-diffusion.  Most  of  the  network  “intelligence”  is  programmed  into  the 
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first  tier.  Aggregation  filters  that  are  determined  by  the  full  nodes  are 
programmed  onto  the  motes  over  the  air. 

2.5  Geographical  and  Energy  Aware  Routing  (GEAR) 

SCADDS  designed  and  studied  (through  simulation)  the  proposed  Geographic 
and  Energy  Aware  Routing  (GEAR)  algorithm,  which  uses  energy  aware  and 
geographically  informed  neighbor  selection  to  route  a  packet  towards  the  target 
region.  This  strategy  attempts  to  balance  node  energy  consumption  and  thereby 
increase  network  lifetime.  Within  a  region,  it  uses  a  recursive  geographic 
forwarding  technique  or  a  restricted  flooding  algorithm  to  disseminate  the  packet. 

SCADDS  simulated  the  above  algorithm  for  uniform  and  non-uniform  traffic 
distributions,  and  compared  its  performance  to  a  non  energy-aware  geographic 
routing  algorithm,  GPSR.  For  non-uniform  traffic,  GEAR  delivers  70%  to  80% 
more  packets  than  GPSR.  For  uniform  traffic,  GEAR  successfully  delivers  25  - 
35%  times  more  packets  than  GPSR.  Hence,  GEAR  exhibits  more  gain  in  non- 
uniform  traffic  scenarios  than  uniform  traffic  scenarios.  The  explanation  is  that 
when  traffic  sources  are  clustered  together,  GEAR’S  energy  balancing  efforts  pay 
off  most.  However,  in  both  cases,  the  GEAR  algorithm  performs  better  in  terms  of 
connectivity  after  partition  (i.e.,  how  routing  the  given  traffic  patterns  affects  the 
rest  of  the  network)  and  the  number  of  normalized  traffic  pairs  broken  down  per 
delivered  packet. 

SCADDS  studied  how  the  protocol’s  performance  being  sensitive  to  imprecise 
neighbor  information.  Simulation  results  show  that  it  is  not  necessary  to  update 
neighbor  information  for  every  packet.  With  increasing  update  threshold,  the 
protocol  performance  degrades  gracefully,  but  the  number  of  control  packets 
generated  drops  dramatically. 

SCADDS  currently  implement  a  prototype  of  GEAR  protocol  in  a  moderate  size 
testbed,  and  are  going  to  test  it  in  real  world  environment.  The  team  is  also  in  the 
process  of  porting  GEAR  to  the  diffusion  implementation  in  ns-2.  On  the  other 
hand,  not  all  applications  always  require  the  same  service  in  terms  of  delay, 
quality  of  data,  or  energy  cost.  For  instance,  most  of  the  time,  sensor  net 
publications  would  trade  delay  for  energy  efficiency.  However,  sometimes,  it  may 
be  concerned  more  about  fast  response  than  energy  cost  or  the  quality  of  data. 
There  is  no  previous  QoS  work  in  sensor  network  context.  Although  there  have 
been  lots  of  QoS  research  for  the  Internet,  the  unique  characteristics  and 
constraints  in  sensor  net  pose  new  challenges,  and  make  the  problem  somewhat 
different  from  its  counterpart  in  the  Internet  domain. 

First  of  all,  because  of  its  stringent  energy  constraint  a  sensor  net  tends  to  be 
more  energy  constrained  than  bandwidth  constrained.  This  makes  admission 
control,  scheduling,  queue  management  not  the  focus  of  ToS  in  sensor  net  any 
more,  although  it  may  still  remain  an  issue  in  some  sensor  net  application 
context.  Second,  Because  of  its  scarce  energy  resources,  sensor  networks  cannot 
afford  over-provisioning  or  brute-force  approaches.  Thirdly,  deployment  in  large 
numbers  requires  the  algorithm  to  be  fully  distributed,  the  question  is  how  to 
achieve  end-to-end  flow  characteristics  with  a  distributed  local  algorithm. 

Before  jumping  into  any  solution,  SCADDS  used  a  simple  analytical  model  to 
show  that  the  gain  of  ToS  is  maximized  when  sensor  network  is  densely  deployed, 

7 


thus  enabling  multiple  alternative  paths;  when  the  delay  sensitive  traffic  and 
regular  traffic  is  mixed,  especially  when  delay  sensitive  traffic  is  only  a  small 
percentage  of  total  traffic,  so  that  all  of  them  can  be  accommodated  on  the  paths 
that  can  satisfy  the  delay  requirements;  and  when  delay-sensitive  applications 
have  strict  delay  requirements.  The  team  is  going  to  study  the  design  space  of 
ToS  in  sensor  networks,  and  design  algorithms  that  can  satisfy  application 
requirements  and  conserve  network  energy  at  the  same  time. 

Previously,  SCADDS  tested  Geographical  and  Energy  Aware  Routing  (GEAR)  in  a 
high  level  simulator,  which  ignored  many  low  level  details.  In  order  to  test  the 
protocol  with  more  realistic  lower  layer  details,  the  team  implemented  and 
evaluated  the  protocol  in  ns-2.  In  the  ns  simulation,  the  team  validated  the  two 
major  design  choices  of  GEAR,  i.e.,  load  balancing  and  recursive  forwarding 
inside  the  target  region. 

To  evaluate  load  balancing  in  GEAR,  SCADDS  compares  GEAR  with  GPSR,  where 
next  hop  selection  is  completely  based  on  geographical  distance,  therefore  no  load 
balancing  is  involved.  In  the  traffic  scenario,  the  team  tested  (1  traffic  pair  with 
source  and  target  randomly  selected,  which  stands  for  non-uniform  traffic 
distribution),  GEAR  can  deliver  20%  to  60%  more  packets  than  GPSR  before  the 
network  is  partitioned.  Moreover,  compared  to  GPSR,  GEAR  extends  network 
lifetime  in  up  to  150%  for  small  packets  (180B). 

Load  balancing  in  GEAR  maximizes  its  gain  with  small  packets  as  explained.  The 
threshold  triggered  beacon  update  will  generate  more  overhead  than  GPSR.  On 
the  other  hand,  in  order  to  do  perimeter  forwarding,  GPSR  carries  extra  fields  in 
its  header.  For  small  packets,  GEAR’S  extra  beacon  overhead  and  GPSR’s  packet 
overhead  will  cancel  each  other  out,  so  that  load  balancing  in  GEAR  will  show  its 
gain.  However,  for  larger  packets,  the  extra  header  fields  in  GPSR  can  be  ignored, 
thus  the  beacon  overhead  will  offset  the  benefits  of  load  balancing  in  GEAR. 
However,  SCADDS  expects  sensor  network  traffic  to  consist  of  mostly  packets  of 
up  to  a  few  hundred  bytes  due  to  its  energy  constraints.  Furthermore,  the 
beacons  in  the  routing  layer  can  be  reduced  by  piggybacking  them  in  data 
packets  or  in  MAC  layer  control  packets. 

SCADDS  also  used  ns  simulations  to  study  if  recursive  forwarding  provides  any 
gain  over  controlled  flooding  in  the  target  region  and  how  this  gain  changes  with 
packet  length.  Unicast  packet  has  RTS/CTS/ACK  broadcast  packets  overhead. 
However,  if  packets  are  large  enough,  so  that  control  overhead  can  be  ignored, 
unicast  recursive  forwarding  is  still  more  efficient  than  flooding  in  the  target 
region.  Simulation  results  show  that  recursive  forwarding  delivers  from  125%  to 
150%  more  packets  than  controlled  flooding  inside  the  target  region.  When  MAC 
control  overhead  is  ignored  (large  data  packets),  recursive  forwarding  exhibits 
major  gains  over  controlled  flooding. 

3.  Sensor-MAC  Performance  in  Multi-hop  Networks 

S-MAC  is  a  MAC  protocol  specifically  designed  for  wire-  less  sensor  networks 
developed  by  the  SCADDS  project.  Building  on  contention-based  protocols  like 
802.11,  S-MAC  strives  to  retain  the  flexibility  of  contention-based  protocols  while 
improving  energy  efficiency  in  multi-hop  networks.  S-MAC  includes  approaches 
to  reduce  energy  consumption  from  all  the  major  sources  of  energy  waste:  idle 
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listening,  collision,  overhearing  and  control  overhead. 

To  measure  the  performance  of  S-MAC  in  multi-hop  networks,  the  project 
measured  the  energy  consumption,  latency  and  throughput  of  S-MAC  in  a  10- 
hop  network  where  nodes  are  placed  along  a  line  with  the  first  node  being  the 
source  and  the  last  one  being  the  sink. 

S-MAC  introduces  periodic  sleep  on  each  node  where  neighboring  nodes 
coordinate  on  their  sleep  schedules.  The  energy  consumption  can  be  greatly 
reduced  by  avoiding  long-time  idle  listening.  However,  latency  will  be  increased 
on  each  hop  due  to  the  sleep.  One  improvement  SCADDS  has  made  over  the 
original  S-MAC  is  to  utilize  an  adaptive  listening  on  each  node.  The  basic  idea  is 
as  follows.  If  a  node  overhears  an  RTS  or  CTS  from  one  of  its  neighbors,  it  will 
wake  up  for  a  short  period  of  time  at  the  end  of  the  transmission.  So  if  the  node 
is  the  next-hop  node,  its  previous-hop  neighbor  will  be  able  to  pass  data  to  it 
without  being  delayed  by  the  normal  sleep  schedule. 

To  compare  the  latency  in  different  configurations,  SCADDS  has  made  S-MAC  to 
run  in  different  modes.  The  first  mode  is  to  let  each  node  strictly  follow  its  sleep 
schedule.  In  the  second  mode,  each  node  follows  its  sleep  schedule  but  adaptively 
listens  at  the  end  of  its  neighbors’  transmissions.  The  third  mode  completely 
disables  the  periodic  sleep,  so  that  there  is  no  sleep-introduced  delay  on  each 
hop. 


Average  message  latency  under  the  lowest  traffic  load 


Figure  2  shows  the  measured  average  latency  on  each  packet  in  the  lowest  traffic 
load.  Compared  with  the  mode  that  each  node  strictly  follows  its  sleep  schedule 
(no  adaptive  listen),  the  mode  with  adaptive  listen  significantly  reduced  the 
latency  in  the  multi-hop  network. 

SCADDS  has  also  measured  the  energy  consumption  and  throughput  in  this  10- 
hop  network.  The  results  of  energy  consumption  basically  conform  to  the  results 
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the  team  obtained  in  the  INFOCOM  paper.  Adaptive  listen  improves  the  energy 
efficiency  in  high-traffic  load.  It  also  improves  the  throughput  in  all  traffic 
conditions  compared  with  the  mode  without  adaptive  listen. 

During  the  S-MAC  measurement,  the  team  also  improved  the  robustness  of  node 
synchronization  in  multi-hop  networks.  Specifically,  SCADDS  added  new 
mechanisms  for  nodes  to  discover  other  nodes  that  follow  different  schedules. 
The  team  is  currently  preparing  a  journal  paper  including  all  the  improvement 
and  new  results  of  S-MAC. 

4.  Application  Techniques 

4.1  Clustering 

Early  SCADDS  work  explored  adaptive  clustering  based  techniques  for  an  object 
tracking  sensor  network.  In  this  application,  sensors  monitor  location  and  status 
of  various  tagged  objects  (e.g.,  projectors,  cameras).  Queries  from  users  about 
these  tagged  objects  are  efficiently  resolved  through  the  proposed  schemes. 

4.2  Aggregation 

SCADDS  examined  the  problem  of  aggregation  of  data  as  it  propagates  down 
established  gradients.  Aggregation  can  be  expressed  as  local  transformation  of 
data  to  reduce  overall  power  consumption.  Hence,  it  might  take  various  forms: 
from  compressing  data,  to  merging  data  temporally  or  spatially  (i.e.  data  coming 
from  different  neighbors),  to  only  forwarding  deductions  on  the  basis  of  received 
data. 

Of  particular  interest  is  the  approach  currently  being  pursued  to  realize  adaptive 
aggregation  through  the  use  of  reinforcement.  In  particular,  Amit  is  investigating 
an  approach  in  which  as  data  aggregates  in  a  node,  that  node  reinforces  the 
strength  of  its  interests  accordingly  and  thereby  pulls  down  additional  data. 
Over  time,  aggregation  points  will  draw  additional  data  toward  them,  increasing 
their  aggregation  capability. 

Preliminary  results  indicate  that  diffusion  can  achieve  significant  energy  savings 
even  with  the  simplest  form  of  application-level  data  processing  (i.e.,  duplicate 
suppression)  in  the  network. 

In-network  data  aggregation  is  essential  for  wireless  sensor  networks  where 
resources  (e.g.,  bandwidth,  energy)  are  limited.  In  the  previous  approach,  data  is 
opportunistically  aggregated  at  intermediate  nodes  on  a  low-latency  tree,  which 
may  not  necessarily  be  energy  efficient.  A  more  energy-efficient  tree  is  a  greedy 
tree,  which  can  be  incrementally  constructed  by  connecting  each  source  to  the 
closest  point  of  the  existing  tree.  The  current  scheme  (a  greedy  approach) 
constructs  a  greedy  aggregation  tree  to  improve  path  sharing. 

The  instantiation  of  directed  diffusion  described  in  the  earlier  work  establishes 
low-latency  paths  between  sources  (sensor  nodes  that  detect  phenomena)  and 
sinks  (user  nodes)  using  only  localized  algorithms.  Paths  from  different  sources  to 
a  sink  form  an  aggregation  tree  rooted  at  the  sink.  Data  from  different  sources  is 
opportunistically  aggregated.  Whenever  similar  data  happens  to  meet  at  a 
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branching  node  in  the  tree,  the  copies  of  similar  data  are  replaced  by  a  single 
message.  Energy-wise,  opportunistic  aggregation  on  a  low-latency  tree  is  not 
optimal  because  data  may  not  be  aggregated  (or  reduced)  near  the  sources. 
SCADDS  proposes  using  a  greedy  incremental  tree  (GIT)  to  improve  path  sharing 
for  more  energy  savings.  The  team  has  implemented  this  greedy- tree  approach  in 
ns-2  and  will  compare  it  to  the  prior  opportunistic  approach  (Section  3).  Due  to 
space  constraints,  some  of  the  more  detailed  algorithms,  simulations,  and 
analysis  have  been  omitted.  Please  refer  to  [Intanagonwiwat02a]  for  more  details. 
The  preliminary  results  suggest  that,  under  investigated  scenarios,  greedy 
aggregation  can  achieve  up  to  45%  energy  savings  over  opportunistic  aggregation 
in  high-density  networks  without  adversely  impacting  latency  or  robustness. 

4.3  Topology  Control 

With  GAF,  Geographic  Adaptive  Fidelity,  [XuOla]  nodes  that  are  redundant  for 
communication  as  determined  by  geographical  position  turn  off  their  radios  in 
order  to  save  energy.  Nodes  alternate  having  their  radios  on  in  order  to 
accomplish  load  balancing.  GAF  uses  location  information  and  an  idealized  radio 
model  to  determine  node  equivalence.  Location  information  may  be  provided  by 
GPS  or  other  location  systems  under  development. 

For  the  initial  discussion,  one  assumes  that  there  is  no  error  in  the  location 
information.  Even  with  location  information  it  is  not  trivial  to  find  equivalent 
nodes  in  an  ad  hoc  network.  Nodes  that  are  equivalent  for  communication 
between  one  pair  of  nodes  may  not  be  equivalent  for  communication  between  a 
different  pair  of  nodes.  GAF  addresses  this  problem  by  dividing  the  whole  area 
where  nodes  are  distributed  into  small  "virtual  grids". 

A  virtual  grid  is  defined  as  follows:  for  two  adjacent  virtual  grids  A  and  B,  all 
nodes  in  A  can  communicate  with  all  nodes  in  B  and  vice  versa.  Thus,  in  each 
grid  all  nodes  are  equivalent  for  routing.  For  example,  Figure  3  shows  three 
virtual  grids,  A,  B,  and  C.  According  to  the  definition  of  virtual  grids,  node  1  can 
reach  any  of  nodes  2,  3,  or  4,  and  nodes  2,  3,  and  4  can  all  reach  node  5. 
Therefore  nodes  2,  3,  and  4  are  equivalent  and  two  of  them  can  go  to  sleep.  The 
team  sizes  the  virtual  grid  based  on  the  nominal  radio  range  R,  farthest  possible 
distance  between  two  nodes  in  adjacent  grids  (since  they  must  be  able  to 
communicate).  If  a  virtual  grid  is  a  square  with  r  units  on  a  side,  then  the  longest 
possible  distance  between  nodes  in  adjacent  grids  is  the  length  of  the  long 
diagonal  connecting  the  two  grids. 


Figure  3 :  A  Virtual  Grid 


In  many  settings,  such  as  indoors  or  under  trees  where  GPS  does  not  work, 
location  information  is  not  available.  The  dependency  on  global  location 
information  thus  limits  GAF's  usefulness.  In  addition,  geographic  proximity  does 
not  always  lead  to  network  connectivity.  GAF  must  make  very  conservative 
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connectivity  assumptions  because  it  guesses  at  connectivity  (based  on  a  radio 
model)  instead  of  directly  measuring  it.  Being  conservative  requires  more  nodes 
to  stay  active  than  necessary,  leading  to  less  energy  conservation.  This  motivates 
Cluster-based  Energy  Conservation  (CEC),  which,  unlike  GAF,  does  not  rely  on 
location  information.  Further,  CEC  itself  directly  and  adaptively  measures 
network  connectivity  and  thus  can  find  network  redundancy  more  accurately  so 
that  more  energy  can  be  conserved. 

CEC  organizes  nodes  into  overlapping  clusters  that  are  interconnected  to  each 
other  by  gateway  nodes. 
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Figure  4:  Compares  GAF  and  CEC  performance 
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5.  Infrastructure 


5.1  Testbed  Development 

SCADDS  experimented  with  several  testbeds  over  the  course  of  the  project.  Its 
main  testebed  was  a  network  of  16  PC/ 104s.  Several  radios  have  been  examined, 
including  the  Radiometrix  in  RPC  and  Berkley  Motes  with  RFM  radio. 

Radiometrix  radios  had  the  advantage  of  including  a  high-level  packet  controller. 
While  helpful  to  getting  the  testbed  going,  lack  of  control  over  the  MAC  protocol 
proved  limiting.  Therefore  the  project  adopted  a  Berkely  mote  as  the  "network 
interface  card"  for  the  testbed  and  developed  the  S-MAC  protocol  (described  in 
Section  xxx). 


The  project  also  used  a  testbed  of  8  WINSng  1.0  computers  running  Windows  CE, 
later  replaced  with  8  WINSng  2.0  computers  running  Linux  with  integrated  radios. 


WINSng  1.0  WINSng  2.0  PC/104  Stayton 


32  bit  platforms 


UCB  Rene  Mote 


8bit  Platform 

Figure  6:  Hardware  platforms  supporting  diffusion 

5.2  Testbed  -  Implementation  and  Analysis 
5.2. 1  Connectivity  Measurements  for  SCADDS  Testbed 

Despite  of  a  simple  and  elegant  formula  in  theory,  the  propagation  property  of  RF 
radio  signal  is  hardly  possible  to  capture  in  real  world.  Fluctuation  and 
Asymmetry  in  connectivity  are  two  major  challenges  when  designing  RF  wireless 
ad  hoc  network  systems.  The  experiments  on  several  directed  diffusion  routing 
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applications  indicate  that  the  connectivity  between  testbed  nodes  may  change 
dramatically  over  long  time  frame.  It  is  very  necessary  to  study  the  connectivity 
and  its  fluctuation  in  the  testbed.  Such  study  would  improve  the  experimentation 
methodology  and  the  understanding  of  wireless  sensor  network  design  in  general. 

5.2.2  Experiment  Details 

The  testbed  consists  of  14  nodes  distributed  over  two  floors  of  an  office  building. 
Each  node  is  equipped  with  a  PC  104  module  and  a  PRC  radio  transceivers.  The 
MAC  protocol  is  a  (simple)  version  of  CSMA/ CA.  Beside  of  the  wireless 
communication  channel,  the  testbed  also  have  wired  infrastructure  to  interface 
itself  to  the  Internet  for  debugging  and  management. 

N 

The  quality  of  a  link  from  node  A  to  B  are  defined  as  throughput  t  =  — - ,  where 

Nt  is  the  number  of  packets  sent  by  A  and  Nr  is  the  packets  received  by  B. 
Without  loss  of  generality,  packets  are  sent  at  the  rate  of  20  per  second,  and  t  is 
computed  based  on  a  10  second  test,  i.e.  around  200  pkt  sent.  The  measurement 
is  logged  to  a  central  server  via  the  wired  infrastructure.  The  experiment  accesses 
the  quality  of  different  links  in  a  round  robin  fashion:  Each  node  in  turn  sends 
out  packets  as  the  sole  transmitter  and  all  the  rest  of  the  nodes  receives.  In  this 
way,  one  can  correlate  the  fluctuations  of  different  links  at  approximately  the 
same  time  if  necessary.  Taking  account  of  the  bookkeeping  time  as  well  as 
cleanup  time  to  make  sure  all  nodes  are  started  in  right  sequence,  each  round  for 
14  node  testbed  lasts  around  9  minutes. 

The  whole  experiment  lasts  45  hours,  which  is  long  enough  to  cover  one  real  life 
cycle  i.e.  one  business  day  in  the  offices.  It  is  quite  interesting  to  see  if  the 
fluctuation  can  be  correlated  to  this  cycle.  For  each  source  and  sink  pair,  the 
quality  log  consists  of  300  points. 

5.2.3  Results 

Figure  [7]  shows  the  connectivity  from  A  to  B  as  the  average  of  all  300 
measurements.  To  depict  the  connectivity  between  different  nodes,  the  team 
visualizes  the  topology  with  different  parameters.  SCADDS  plots  those  links  with 
connectivity  greater  than  certain  threshold,  showing  how  well  the  network  is 
connected.  With  even  very  high  threshold  0.9,  the  network  is  still  connected.  With 
threshold  0.75,  the  overlay  topology  is  rich  enough  to  do  multiple  hop  wireless 
communication  experiments. 

SCADDS  is  interested  in  those  links  with  heavy  loss  but  positive  connectivity 
because  in  general  the  source  and  sink  usually  cross  multiple  "hops"  and  data 
can  be  delivered  over  those  links  with  less  delay  but  less  reliable  than  over  those 
good  single  hop  links.  It  seems  that  lossy  links  are  quite  common  in  this  network. 
For  Example,  the  links  with  less  than  0.1  are  spread  over  most  portion  of  the 
network. 
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Connectivity  Changes  Over  Time  are  Unpredictable 


5.3  Software  Developments 

Software 

•  Radiometrix  Device  Drivers 

This  software,  developed  as  part  of  the  SCADDS  project,  was  released  to  the 
public  in  June  of  2000;  for  more  information  see: 

<http:  /  /  www.  circlemud  .org/ ~j  elson  /  software /radiometrix> 

This  package  contains  a  Linux  device  driver  for  the  RPC  (Radio  Packet  Controller) 
model  of  radio  manufactured  by  Radiometrix.  The  RPC  is  a  fairlylow-power,  self- 
contained,  short-range,  plug-on  radio.  It  has  been  a  critical  part  of  the  testbed 
infrastructure  for  implementation  and  validation  of  directed  diffusion  and  other 
algorithms. 

•  Emlog 

This  software  was  also  developed  as  part  of  the  SCADDS  project  and  released  to 
the  public  in  June  of  2000;  for  more  information  see: 

<http :  /  /  www.  circlemud .  org  /  ~j  elson  /  software  /  emlog> 

Emlog  is  a  Linux  kernel  module  that  makes  it  easy  to  access  the  most  recent  (and 
only  the  most  recent)  output  from  a  process.  It  works  just  like  "tail  -f  on  a  log 
file,  except  that  the  storage  required  never  grows.  This  is  veiy  important  for  the 
logging  and  debugging  facilities  in  embedded  systems  where  there  isn’t  enough 
memory  or  disk  space  for  keeping  complete  log  files,  but  the  most  recent 
debugging  messages  are  sometimes  needed  (e.g.,  after  an  error  is  observed). 
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S-MAC  Communication  Stack  on  Mica  Motes 


SCADDS  developed  a  radio  communication  stack  on  the  Mica  Motes  running 
TinyOS  ,  developed  by  University  of  California,  Berkeley.  It  is  an  alternative  of  the 
standard  communication  stack  in  the  TinyOS  release.  It  provides  new  features 
such  as  flexible  packet  format  and  headers,  reliable  transmission  of  variable 
length  packets  and  the  full  S-MAC.  Current  releases  are  at 
<http://www.isi.edu/ilense/software/ smac/ index.html> 

•  Directed  Diffusion 

SCADDS  made  a  number  of  releases  of  directed  diffusion  over  the  course  of  the 
project,  including  support  for  Linux,  WINSng  2.0,  and  ns-2.  Current  releases  are 
at  <http: / /www.isi.edu /flense / software/ diffusion/ index.html> 

5.4  SensIT  Collaboration/Integration 

Over  the  course  of  the  project  the  SCADDS  group  participated  actively  in  SensIT 
integration  activities.  Examples  include  supporting  field  tests  at  SITEXOO  and 
SITEX01,  both  of  which  used  directed  diffusion.  They  worked  with  MIT-LL  to  help 
define  the  diffusion  API  and  insure  that  the  MIT-LL  and  ISI  implementations  were 
compatible.  They  worked  with  with  researchers  at  MIT-LL,  Cornell,  PARC,  BAE 
Systems,  PSU  and  other  institutions  to  support  technology  transfer  of  directed 
diffusion  and  other  protocols. 

6.  Personnel 

Over  the  course  of  the  SCADDS  project,  the  following  personnel  were  involved: 

Staff  at  USC/ISI: 

•  Deborah  Estrin 

•  Padmapama  Haidar 

•  John  Heidemann 

•  Ramesh  Govindan 

•  Fabio  Silva 

•  Wei  Ye 

•  Jong-Suk  Ahn 

•  Cengiz  Alaettinoglu 

Graduate  Students  at  USC/ISI: 

•  Nirupama  Bulusu 

•  Vladimir  Bychkovskiy 

•  Jerry  Elson 

•  Deepak  Ganesan 

•  Lewis  Girod 

•  Chalermek  Intanagonwiwat 

•  Amit  Kumar 

•  Satish  Kumar 

•  Kun-chan  Lan 

•  Fred  Stann 
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•  YaXu 

•  Jerry  Zhao 

•  Yen  Yu 

Collaborators: 

•  Phillippe  Bonnet  (DIKU) 

•  Professor  Culler  (UC  Berkeley) 

•  Julia  Liu  (PARC) 

•  Ted  Faber  (USC/ISI) 

•  Richard  Muntz  (UCLA) 

•  Professor  Pister  (UC  Berkeley) 

•  Jim  Reich  (PARC) 

•  Mani  Srivastava  (UCLA) 

•  Feng  Zhao  (PARC) 

•  Richard  Brooks  (PSU) 

•  Joe  Reynolds  (?) 
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